---
layout: minimal
title: "0.2"
description: "Concurrent indexing, cursor pagination, and more"
# authors:
#   - "[Kevin](https://twitter.com/typedarray)"
date: "March 15, 2024"
---

import { Footer } from "../../components/footer";

# 0.2 [Concurrent indexing, cursor pagination, and more]

<div className="flex flex-row items-center gap-4 mt-5">
  <img src="/kevin-avatar.jpg" className="w-10 h-10 rounded-full" />
  <div className="flex flex-col">
    <span className="font-medium">Kevin</span>
    <span className="text-sm" style={{ color: "var(--vocs-color_text3)" }}>March 15, 2024</span>
  </div>
</div>

Ponder 0.2 adds concurrent indexing, cursor pagination in GraphQL, and a new column type for hexadecimal strings.

Before this release, Ponder's indexing engine processed events indexing sequentially (one event at a time). But in theory, depending on which tables they access, indexing functions can often run in parallel. This release introduces concurrent indexing, which allows the indexing engine to process multiple events at the same time.

For a simple ERC20 app, concurrent indexing was 22% faster than sequential in our benchmark. It varies from app to app, but most Ponder apps have a theoretical speedup of 20-50% with concurrent indexing.

In many cases, indexing functions do not depend on the results of other indexing functions. Consider a simple Ponder app that indexes ERC20 Transfer events.

:::code-group

```ts [ponder.schema.ts]
import { createSchema } from "@ponder/core";

export default createSchema((p) => ({
  TransferEvent: p.createTable({
    id: p.string(),
    from: p.hex(),
    to: p.hex(),
    amount: p.bigint(),
    timestamp: p.int(),
  }),
}));
```

```ts [src/index.ts]
import { ponder } from "@/generated"

ponder.on("ERC20:Transfer", async ({ event, context }) => {
  const { TransferEvent } = context.db;

  await TransferEvent.create({
    id: event.log.id,
    data: {
      from: event.args.from,
      to: event.args.to,
      amount: event.args.value,
      timestamp: event.block.timestamp,
    }
  });
});
```

:::

In this example, the indexing function only writes to the `TransferEvent` table. It doesn't read from any tables.

:::info
  Technically, the indexing engine still uses a finite concurrency factor that's
  often less than the theoretical concurrency. In these cases, indexing
  throughput is bottlenecked by the database, so the internal queue concurrency
  starts to matter less.
:::

Concurrent indexing is backwards compatible and requires no changes to your indexing function code.

### How it works

Ponder's build step uses static analysis to determine which tables an indexing function reads from and writes to, and uses that to construct an indexing function dependency graph. Armed with the dependency graph, the indexing engine enqueues events to be processed as soon as their dependencies are met. The static analysis step handles most common code organization patterns, but it's not perfect. If static analysis fails for any reason, it falls back to sequential indexing.

To check the indexing function dependency graph, enable debug logging (either run `ponder dev -v` or set the `PONDER_LOG_LEVEL` env var to `"debug"`) and you'll see logs like this for each of your indexing functions.

```bash filename="shell"
DEBUG  Registered indexing function BasePaintBrush:Transfer (selfDependent=true, parents=[BasePaint:Painted])
DEBUG  Registered indexing function BasePaint:Started (selfDependent=true, parents=[])
DEBUG  Registered indexing function BasePaint:Painted (selfDependent=true, parents=[BasePaintBrush:Transfer, BasePaint:ArtistsEarned, BasePaint:TransferSingle, BasePaint:TransferBatch])
DEBUG  Registered indexing function BasePaint:ArtistsEarned (selfDependent=true, parents=[BasePaint:Painted, BasePaint:TransferSingle, BasePaint:TransferBatch])
DEBUG  Registered indexing function BasePaint:ArtistWithdraw (selfDependent=false, parents=[])
DEBUG  Registered indexing function BasePaint:TransferSingle (selfDependent=true, parents=[BasePaint:Painted, BasePaint:ArtistsEarned, BasePaint:TransferBatch])
DEBUG  Registered indexing function BasePaint:TransferBatch (selfDependent=true, parents=[BasePaint:Painted, BasePaint:ArtistsEarned, BasePaint:TransferSingle])
```

## Cursor pagination

Before this release, the GraphQL API and the `findMany` database method only supported offset pagination. Offset pagination is simple, but has a number of [well-documented downsides](https://akashrajpurohit.com/blog/navigating-your-database-efficiently-cursor-based-pagination-vs-offset-based). This release adds cursor pagination with support for arbitrary sort orders. Notably, this eliminates the maximum offset of 5,000 records, which makes it possible to efficiently paginate through all records in a table regardless of size.

:::code-group

```graphql [Query]
query {
  persons(orderBy: "age", orderDirection: "asc", limit: 2) {
    items {
      name
      age
    }
    pageInfo {
      startCursor
      endCursor
      hasPreviousPage
      hasNextPage
    }
  }
}
```

```json [Result]
{
  "persons" {
    "items": [
      { "name": "Sally", "age": 22 },
      { "name": "Lucile", "age": 32 },
    ],
    "pageInfo": {
      "startCursor": "MfgBzeDkjs44",
      "endCursor": "Mxhc3NDb3JlLTA=",
      "hasPreviousPage": false,
      "hasNextPage": true,
    }
  }
}
```

:::

Take a look at the new [pagination docs](/docs/query/graphql#pagination) for more details.

## `p.hex()`

In most TypeScript programming environments, it's common to use a hexadecimal string representation for byte arrays like Ethereum addresses. This release adds a new column type, `p.hex()`, which is a more efficient way to store hexadecimal strings in the database.

```ts filename="ponder.schema.ts" {5,9,10,15}
import { createSchema } from "@ponder/core";

export default createSchema((p) => ({
  Account: p.createTable({
    id: p.hex(), // Address
    balance: p.bigint(),
  }),
  Transaction: p.createTable({
    id: p.hex(), // Transaction hash
    blockHash: p.hex(), // Block hash
    // ...
  }),
  Log: p.createTable({
    id: p.string(),
    topic0: p.hex(), // Log topic
    // ...
  }),
}));
```

Under the hood, `p.hex()` uses the `bytea` column type in Postgres and `blob` in SQLite. Database operations using `p.hex()` have similar performance to those using `p.string()`, but `p.hex()` values take up less space in the database.

:::warning
  The `p.bytes()` column type had a serious performance issue when used with
  `id` columns. This has been fixed with the migration to `p.hex()`.
:::

## Get started

To upgrade an existing app, run:

:::code-group

```bash [pnpm]
pnpm upgrade @ponder/core
```

```bash [yarn]
yarn upgrade @ponder/core
```

```bash [npm]
npm upgrade @ponder/core
```

:::

<Footer />